Resource Constrained Video Coding/Adaptation

Resource Constrained Video Coding/Adaptation Yong Wang Typical video coding systems focus on optimization of the video quality subject to certain resource constraints, such as bandwidth. The rate distortion optimization framework has been widely used in optimizing the performance tradeoffs. One fundamental issue involved is the difficulty in estimating the rate-distortion relationships under different design choices. Likewise in a video adaptation system, where encoded videos are adapted to meet dynamic resource requirements, the relationships between resources and video quality are difficult to estimate analytically. In this thesis, we extend the rate-distortion concepts to a broader framework based on utility function (UF), which models the relations among resources, video utility, and adaptation operations. We then propose a novel content-based paradigm to automatically predict the utility function of a video and determine the optimal video adaptation operation. Our approach is flexible in that different types of utilities, resources, or operations can be easily incorporated. Our content-based prediction approach is shown to be accurate. In the second half of the thesis, we further extend the framework to investigate the influence of power resource constraint. We develop a joint power-rate-distortion optimization method to achieve optimal video quality under joint constraints of available bandwidth and power. Video adaptation modifies an existing encoded video stream to different frame rates, spatial resolutions, or others forms to meet dynamically changing conditions of networks or client platforms. Multiple dimensional adaptation (MDA) can be constructed by combining more than one type of adaptation operations. Nevertheless, selection of the best MDA operation among various choices is often done in an ad hoc way and hard to generalize. To provide a systematic solution, we present a framework based on a general utility function, which models the relationships among video quality, resources, and adaptation operations. We show utility functions are correlated to the content features of the video, which can be automatically extracted in real time. Machine learning methods are then developed to train systems which predicts the best adaptation operation according to the content features. We evaluate the proposed methods over videos compressed by MPEG-4 and motion compensated embedded zeroblock coding (MC-EZBC) respectively. To consider different quality measurements, perceptual subjective evaluation is also conducted, in addition to objective quality measurements. The characteristics of MDA over these codecs are analyzed, the utility functions are constructed, and the performance of content based operation prediction is validated with excellent performance. Complexity aware video coding is important for mobile/wireless devices where the computational capability or power resource is restricted. Emerging video coding standards like H.264 achieve significant improvements in video quality, at the expense of greatly increased computational complexity at both the encoder and the decoder. We propose a systematic solution for complexity optimized video decoding by extending the conventional rate-distortion framework. A complexity-control algorithm is also developed to meet the overall specified target complexity level and keep the complexity consistent significant complexity reduction (up to 60% in motion compensation) with little quality degradation (less than 0.3dB).